label search altern activ learn appli human resourc build classif model extrem class imbal josh attenberg foster provost paper analys altern techniqu deploi low cost human resourc data acquisit classifi induct domain exhibit extrem class imbal tradit label strategi activ learn ineffect problem build classifi brand control content adjac line advertis frequent worri advertis objection categori rare distribut impress encount line advertis rare tradit sampl techniqu find posit exampl train effect model altern deploi human resourc train data acquisit guid learn search explicitli train exampl class show extrem skew basic techniqu guid learn complet domin smart activ strategi appli human resourc select case label critic rel cost search versu label demonstr tradeoff rel cost show cost skew set choic search activ label equivoc hybrid strategi combin benefit 